System and method for augmentation of endoscopic surgery

ABSTRACT

The present method and apparatus use image processing to determine information about the position of a designated object. The invention is particularly useful in applications where the object is difficult to view or locate. In particular, the invention is used in endoscopic surgery to determine positional information about an anatomical feature within a patient&#39;s body. The positional information is then used to position or reposition an instrument (surgical instrument) in relation to the designated object (anatomical feature). The invention comprises an instrument which is placed in relation to the designated object and which is capable of sending information about the object to a computer. Image processing methods are used to generated images of the object and determine positional information about it. This information can be used as input to robotic devices or can be rendered, in various ways (video graphics, speech synthesis), to a human user. Various input apparatus are attached to the transmitting or other used instruments to provide control inputs to the computer.

BACKGROUND OF THE INVENTION

1. Field of the Invention

This invention relates to the field of endoscopic surgery. Morespecifically the invention relates to obtaining accurate positionalinformation about an anatomical structure within a patient's body andusing this information to accurately position endoscopic cameras andsurgical instruments within the patient's body.

2. Description of the Prior Art

Systems have been developed to augment a human surgeon's ability toperform surgery on a patient by providing the surgeon withintraoperative images of anatomical structures within the patient'sbody. Typically, these systems comprise a surgeon specialized form ofcamera or medical telescope. Further, a class of these systems, whichincludes endoscopic and laparoscopic instruments, has reduced theinvasive nature of many surgical procedures.

This class of systems has two salient characteristics in common: First,the surgeon using the system cannot directly manipulate the patient'sanatomy with his fingers, and second, the surgeon cannot directlyobserve what he is doing. Instead, the surgeon must rely on instrumentsthat can be inserted through a trocar or through a working channel of anendoscope. Often, since his hands and attention are fully occupied inperforming the procedure, the surgeon must rely on an assistant to pointthe endoscopic camera while the surgery is performed.

To ameliorate the awkwardness of this arrangement, robotic augmentationdevices have been developed for endoscopic surgery. One such device isdescribed in detail in a copending application entitled "System andMethod for Augmentation of Surgery" Ser. No. 07/714,816 filed Jun. 13,1991 which is herein incorporated by reference.

Robotic augmentation devices can potentially greatly assist surgeonsduring an operation. Robotic devices do not fatigue. Potentially, theycan position medical telescopes and surgical instruments very accuratelyand can perform precise repositioning and repetitive functions. However,in order for these advantages to be realized, a number of problems needto be solved. The surgeon still needs to determine what motions therobotic device is to make and requires a means to communicate with thecomputer controlling the robot. In a few cases, such as orthopaedicmachining of bone or pre-planned excision of a tissue volume determinedfrom preoperative medical images (such as CT or MRI scans), thesemotions may be pre-planned. However, in other cases, the surgeon needsto directly observe the patient's anatomy and interactively specify themotions to be made relative to anatomical features and the medicaltelescopes. In these cases, means of accurately locating anatomicalfeatures and instruments relative to the medical telescopes and to eachother and of using this information to control the robotic augmentationaids are necessary.

A specialized robotic device for stepping a resectoscope through apreprogrammed sequence of cuts in thranurethral prostatectomies has beendeveloped. However, this system does not address the problem ofproviding the surgeon with a convenient means of controlling the viewavailable through an endoscopic device or of providing the surgeon withmeans of interactively manipulating surgical instruments in response tointraoperative imaging and other sensory information.

There has been one attempt to provide voice control of a flexibleendoscope in which servomotors attached directly to the control knobs ofa commercial flexible endoscope were activated in response to voicecommands by the surgeon. Difficulties of this approach include: (a) thesurgeon (or an assistant) must still determine which direction todeflect the endoscope tip to provide a desired view and, consequently,must keep track of the relationship between the endoscope tip and theanatomical structures being observed; (b) these corrections must be madecontinually, distracting the surgeon from more important matters; and(c) the use of voice commands for this purpose is subject to errors,potentially distracting to the surgeon, and may make the use of voicefor communication between the surgeon and operating room personnel moredifficult. Several research efforts are directed to providing improvedmechanisms for flexible endoscopes. These devices do not, however,simplify the surgeon's problem of controlling the endoscopic camera toobtain a desired view, either by himself or by communicating with askilled operator.

3. Statement of Problems with the Prior Art

Unfortunately, the medical telescopes which are used in minimallyinvasive surgery have limited fields of view. As a result, only a smallpart of the anatomical Feature hidden inside the patient's body can beviewed at a one time. Furthermore, surgical telescopes typically provideonly a single vantage point at any one time and it is difficult toprovide the desired view.

Normally, to compensate for this limited field of view, a surgicalassistant operates the telescope, reorienting it to produce many viewsof the anatomical feature. While doing this, the assistant mustcontinuously keep track of the relative orientation between thetelescope and the patient's anatomy in order to be able to quickly andcorrectly aim the telescope at the surgeon's request. He or she mustalso correctly interpret the surgeon's desires, which are not alwaysevident from the surgeon's verbal comments.

This creates a number of problems. Surgical procedures of this naturenow require an additional highly-skilled person to assist the surgeon inmanipulating the medical telescope because the surgeon is using both ofhis hands performing other tasks. The communication that is requiredbetween the surgeon and the assistant increases the potential for anerror while performing the surgery. The surgeon (and assistant) have todevelop and keep a mental image of the entire hidden anatomical featurebecause the telescope can not capture the full image of the feature.Many telescopes, whether flexible or rigid, provide an oblique view,i.e., the direction of view is not coincident with the main axis of thetelescope. This further exacerbates the difficulties of correctly aimingthe telescope to achieve a desired view and increases the likelihoodthat the surgeon or the assistant could misconstrue the image presentedor lose the orientation of the telescope with respect to the anatomicalfeature. Human fatigue contributes to a degradation of positioning ofthe telescope and/or of the interpretation of the images that thetelescope transmits.

Accordingly, there is a need for a way to obtain accurate and reliableinformation about the position and appearance of anatomical featureshidden within a body. There also is a need for an apparatus toaccurately position and orient surgical instruments and/or medicaltelescopes within a body and to provide accurate information about theirposition with respect to hidden anatomical features. Further, there is aneed to provide a reliable and accurate interface between the surgeonand his surgical instruments so that he can accurately position theseinstruments with respect to an anatomical feature within a body withoutremoving his hands from his instruments.

OBJECTIVES

An objective of this invention is to provide an improved method toobtain and display accurate information about the position of ananatomical feature within a patient's body.

Also an objective of this invention is to provide an improved method ofpositioning endoscopic cameras and other surgical instruments within apatient's body.

A further objective of this invention is to provide an interface for asurgeon to accurately position an endoscopic camera and/or othersurgical instruments within a patient's body without removing his handsfrom the instrument.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 shows a schematic view of a system used for computer augmentationof surgical procedures.

FIG. 2 is a detail of FIG. 1 showing a distal fine motion rotationalmanipulator.

FIG. 3 shows an embodiment of the invention using a stereoscopicvisualization system.

FIG. 4 shows an embodiment of the present invention comprising tworobotic manipulators.

FIG. 5 shows positions in 2D and 3D Cartesian coordinate systems.

FIG. 6 shows the pin-hole mathematical model of a camera.

FIG. 7 shows a method of computing a position in three dimensions usingtwo nonparallel camera vantage points.

FIG. 8 shows the use of passive visual targets to determine a positionof a surgical instrument.

FIG. 9 shows a method of computing a position in three dimensions usingtwo parallel camera vantage points.

FIG. 10 shows a method of using oblique medical telescopes.

SUMMARY OF THE INVENTION

The present invention is a method and apparatus for determiningpositional information about an object and then using this informationto position instruments in relation to the object. The invention hasmany applications but is particularly useful when the object is hiddenfrom view or in a location that is difficult to access. One preferredembodiment, used in endoscopic surgery, determines positionalinformation about a designated anatomical feature which is hidden withina patient's body. The information is used to position surgicalinstruments in the body with respect to the anatomical feature.

The invention first positions an instrument, e.g. a surgical instrumentinserted inside a patient's body, at a desired position relative to adesignated object (anatomical feature). The instrument is capable oftransmitting an image of the object to a computer which then determinespositional information about the object by using various types of imageprocessing. The information is then related to a human (e.g., a surgeon)or to a computer controlling a robotic apparatus. The positionalinformation is used to position or reposition the transmittinginstrument and/or other instruments relative to the designated object.

To further facilitate use of the invention, a number of different outputmodes For conveying information from the imaging instruments andcomputer to humans in the operating room are provided.

To further facilitate use of the invention, input devices areincorporated on the inserted instruments so that a human user can inputrequests to the system while concurrently manipulating the instrument.Other methods of inputting requests to the system, such as voicerecognition systems, are also incorporated so that communications withthe system does not interfere with instrument manipulation.

DETAILED DESCRIPTION OF THE INVENTION

Referring to FIG. 1, there is shown a schematic view of a system for usein computer augmentation of laparoscopic or similar procedures. Thesystem generally comprises a manipulator apparatus or robot 242, acomputer 243, a drive motor interface 244, a monoscopic monitor 247 witha suitable image processor 245 and graphics adaptor 246, a stereoscopicmonitor 272 with suitable stereo display system 271, and a terminal 248for connecting additional input devices to computer 243.

A manipulator similar to the manipulator 242, used in this preferredembodiment, is described in detail in the copending U.S. applicationSer. No. 07/714,816 filed on Jun. 13, 1991.

Referring to FIGS. 1 and 2, the manipulator 242 comprises a proximalrectilinear manipulator 6 and a remote center-of-motion distalmanipulator 240. The proximal manipulator 6 comprises three mutuallyorthogonal sliding motion sections 1, 2, and 3, which provide motion inthe X, Y, and Z directions. Sections 1, 2, and 3 are equipped withcomputer-controlled motorized drives 4 connected to motion interface 244and also have manual locking clamps 5. The remote center-of-motiondistal manipulator 240 comprises rotational sections 7, 250, 251, and252 to provide θ_(p) θ_(x), θ_(y), and distal θ_(z) rotational motion,and a slide motor 253 adapted to axially slide instrument 254. Thesesections are equipped with computer-controlled motorized drives 249interfaced to motor interface 244 and have manual locking clamps 255.Each of the moving sections of manipulator 242 can be actuated eithermanually or under computer control and can optionally be locked by amanual locking device. All the motorized drives 4 and 249 are controlledby computer 243 through motor interface 244.

Referring to FIG. 2, there is shown a schematic view of the distal finemotion rotational manipulator 240 with an instrument 241 insertedthrough an incision into a patient's body. In the embodiment shown, thedistal manipulator 240 provides a five degree-of-freedom (θ_(p), θ_(x),θ_(y), θ_(z), and d) remote center-of-motion wrist, which is supportedby the aforementioned proximal positioning system with three orthogonallinear degrees of freedom (X, Y, and Z). The proximal linear degrees offreedom are used to place the center-of-motion M of the remotecenter-of-motion wrist at the position of insertion into the patient'sbody P. Any alternative mechanical structure (such as a SCARAmanipulator, manufactured and sold by IBM) with sufficient degrees offreedom could be substituted for this purpose.

The four distal revolute degrees of freedom and the sliding degree offreedom of manipulator 240 give the surgeon a five degree-of-freedomspherical work volume centered at the insertion point M. These degreesof freedom may be selectively locked or moved independently (manually orunder computer control) to assist the surgeon in achieving a desiredprecise alignment. Furthermore, small motions within the work volume canbe achieved with only small motions of the individual axes. The point M(i.e., the point at which the surgical instrument enters the patient)remains unaffected by any motions of the distal manipulator 240. Thusthe manipulator may be moved through its work volume without requiringthat the patient position be moved or that the size of the entry woundbe enlarged.

One consequence of this design is that motion of the proximalmanipulator 6 is not needed unless the patient is moved. Consequently,in a preferred embodiment, the motion of proximal manipulator 6 isdisabled by manual locking and/or disabling of drive motors whenever aninstrument is inserted into the patient. In this mode, the controlcomputer 243 interprets commands requesting motion of manipulator 242 asfollows. When a motion is requested, the control computer 243 attemptsto satisfy the request by moving only distal manipulator 240. If themotion can be accomplished in more than one way, the computer selectsthe motion that minimizes the motion of the most proximal revolutemotion section 7 (i.e., it minimizes motion of θ_(p)). If the motioncannot be accomplished perfectly, the computer selects the motion ofdistal manipulator 240 that most closely approximates the desiredmotion. Modes are available to select minimization of positional errorof the tip of instrument 241, orientation error, or weightedcombinations thereof. If the error is greater than a prespecifiedthreshold amount, the control computer notifies the surgeon usingsynthesized speech, an audible alarm, or other means, and makes nomotion unless the surgeon explicitly instructs it to proceed, usingvoice recognition or other input modality. One alternative embodimentmight seek always to minimize the total motion of the distal manipulator240, again forbidding motion of proximal manipulator 6 whenever asurgical instrument held by the distal manipulator is inserted into thepatient's body. Yet another might permit small motions of the proximalmanipulator, so long as the center-of-motion M stays within a specifiedthreshold distance (e.g., 3 mm) of the original value.

If desired, a flexible tip may be added to the distal end of instrument241 to provide additional degrees of freedom. In the case where aviewing instrument such as instrument 254 is used, an additionaldegree-of-freedom in adjusting the gaze direction may be provided byadding an adjustable-angle mirror or prism to the distal end of theinstrument.

Referring again to FIG. 1, the instrument 254, in the embodiment shown,includes a video camera 259 and a light source 277 connected to theinstrument via a fiberoptic cable 278. The video output of the camera259 is fed into the graphics adaptor 246, where it may be optionallymixed with graphics output from computer 243 and displayed on monitor247. The video output from the camera is also optionally fed into theimage processing system 245, which analyzes the image produced by thecamera and provides information to computer 243 about the relativeposition of the surgeon's instruments, the camera, and the patient'sanatomy. The video information from the camera may be also optionallysupplied to the stereo display system 271, which can assemble astereoscopic view of the patient's anatomy from two or more images takenfrom different vantage points and display the image on the stereoscopicmonitor 272.

In one preferred embodiment, the stereo display system is aSTEREOGRAPHICS CRYSTALEYES (trademark of StereoGraphics, Inc.) system,where the two video signals are displayed on a stereoscopic monitorwhich alternatively displays the left and right eye image at a frequencyof 120 Hz, updating the video information for each eye 60 times persecond. The surgeon wears stereoscopic liquid crystal (LC) goggles 273,which are synchronized with the monitor and alternatively block lightfrom entering left and right eye such that the left eye receives onlythe video signal from the left camera and the right eye receives onlythe information from the right camera. The frequency of alternationbetween left and right images is sufficiently high such that the surgeonperceives no flicker but rather a continuous stereoscopic image of thepatient's anatomy. Other stereo display technologies are available andmay be used.

In the embodiment shown, the surgeon is using a second surgicalinstrument 260 inside the patient's body, which has passive visualtargets 276 placed on it. These targets 276 are markings on theinstrument and are chosen so as to be easily locatable by the imageprocessing system 245 in the images supplied by the camera 259.

The set of input/output devices attached to input/output interface 248of computer 243 shown in FIG. 1 may include a computer voice recognitionand synthesis system 267, a joystick 268 mounted on the surgicalinstrument 260 and a sterilized touch screen 269 mounted on monitor 247.In the preferred embodiment the joystick is a small device, functionallyidentical to a 2D or 3D mouse, but designed such that it can be mounteddirectly onto a surgical instrument and such that at least two degreesof freedom of motion can be specified by applying pressure on a smalljoystick protruding from the device. One implementation of such a deviceuses strain gauges to translate an applied pressure or force intoincremental displacement or velocity information. In another embodiment,a six degree-of-freedom input device, such as SPACEBALL (A Trademarkowned by Spaceball Technologies, Inc.) could be used to specify motionin any of the six degrees of freedom. Such a device could be mounted ona surgical instrument, on the manipulator structure, or at any otherconvenient point. One advantage of mounting an input device such as asmall joystick on a surgical instrument is that the surgeon can easilymanipulate the joystick without removing his hands from the surgicalinstrument, thus permitting him to provide information to the computer(for example, of a desired direction of motion of a medical telescope)without interrupting his work.

The speech recognition and synthesis system 267 includes means ofinputting information to the system, such as a (possibly head mounted)microphone 275, and a means of conveying information to the surgeon,such as a speaker 274. The speech recognition system 267 is capable ofunderstanding a vocabulary of instructions spoken by the surgeon and canrelate the information about the commands it has received to thecomputer 243. The surgeon may use any of these modalities, eitherseparately or in combination, to position graphic objects on the monitor247, to select commands or operating modes from menus, and to commandmotions of the manipulator 242.

Referring to FIG. 3, there is shown an alternative embodiment of thesystem for computer augmentation of laparoscopic or similar surgicalprocedures. In this embodiment, the surgical instrument 254a is astereoscopic medical camera, which incorporates two independent lenssystems or optical fibers and is capable of transmitting twosimultaneous images from the patient's body. The two lenses areseparated by a small (known) distance and are thus able to provide astereoscopic image. One embodiment of such a device would comprise twoside-by-side fiberoptic bundles or lens systems and one fiberoptic lightchannel. The assembly would be surrounded by a suitable cylindricalcasing. The video signals from the two cameras 259a and 259b are fedinto the stereo display system 271 and displayed to the surgeon on astereoscopic display monitor 272. Using interface hardware known in theart, both video signals are also optionally supplied to the imageprocessing system 245 and the graphics adapter 246.

Another embodiment of the system is shown in FIG. 4, where the systemcomprises two manipulators 240a and 240b, carrying surgical instruments241a and 241b, respectively. In one embodiment, one of the surgicalinstruments is a medical telescope, whereas the other instrument is asurgical tool, such as medical forceps. Since both instruments areattached to robotic devices, both can be actively positioned undercomputer control. On the other hand, as with the single manipulator armin the case above, either or both robots can be controlled manually byreleasing, adjusting, and relocking joint axes one at a time. In analternative embodiment, both surgical instruments 241a and 241b comprisemedical telescopes or other means of transmitting an image outside of apatient's body. In such an embodiment, one of the instruments (forexample, 241a) may also comprise a surgical tool such as a miniaturizedsurgical forceps. In this case, information from images taken at twovantage points may be combined to provide precise 3D information toassist in placing the surgical instrument precisely on the desiredportion of the patient's anatomy.

Referring again to FIG. 1, the image processing system 245 may be usedto locate features on the patient's anatomy of interest to the surgeon.In this mode, the surgeon would designate a feature of interest by anyof a number of means to be explained below. On the surgeon's command,supplied via any appropriate input device attached to the terminal 248,the computer 243 would instruct the image processing system 245 toacquire an image and precisely locate the designated anatomical feature.In one embodiment, a reference image of the designated Feature would beacquired in response to the surgeon's command and stored. Imagecorrelation techniques would be used to locate the feature duringsurgery. In an alternative embodiment, synthetic reference images couldbe generated from computer reconstructions of preoperative medicalimages and models. Once a feature has been located, the manipulator 242can be moved to place the feature at any desired position in the camerafield of view. If desired, an additional image may be acquired, thefeature re-located, and a further adjustment made to refine the desiredplacement of the camera. This process may be repeated a number of timesto "zero in" on a feature to any desired accuracy. Each of the foregoingsteps is explained below.

As a matter of nomenclature, we will in the following text refer topositional information in a number of ways. Unless otherwise specified,the terms "position" and "location" will be used interchangeably. Wewill be referring to two-dimensional (2D) and three-dimensional (3D)positions. When referring to an image obtained by a single monoscopiccamera, an "image location" or "image position" should be understood asa 2D location within the 2D image. Referring to FIG. 5a, such a locationA (within a 2D image 800) is given as a pair of coordinates (x,t) Whenthe image is stereoscopic, "image location" or "image position" shouldbe understood as a 3D location within the volume of the stereoscopicimage. Referring to FIG. 5b, such a location B is described by a tripleof coordinates (x,y,z). We will also refer to positions of anatomicalfeatures. Such features are part of the patient's anatomy and allreferences to "feature location" or "feature position" should beunderstood as 3D positional information about the feature in question.

In order to use and manipulate images of the patient's anatomy, imagesmust first be acquired. Referring to FIG. 1, this is done by feeding thelive video signal from camera 259 into the image processing system 245comprising at least one video digitizer. A video digitizer is a devicecapable of converting an analog video signal into a digital signal,which can be stored in computer memory and arbitrarily modified by thecomputer. Conversely, a video digitizer can also convert a digitized(and possibly modified) video signal back into analog form for displayon a standard monitor.

If positional information is to be extracted from images obtained by acamera/lens system, a mathematical model of the camera and the lens mustbe available to relate image points (i.e., points on the camera'simaging plane) to the corresponding world points (i.e., 3D locations inthe actual environment). To a good approximation, a perfect camera/lenssystem can be modeled as a pin-hole system, illustrated in FIG. 6. Thefigure depicts a camera with a lens 600 positioned a distance f in frontof the image plane 601. The quantity f is referred to as the focallength of the lens. A point W=(x,y,z) lying in the plane 602 a distanced=-z in front of the lens is imaged onto the image plane 601 tit thelocation C=(x',y'), where x/d=x'/f and y/d=y'/f.

Given the image coordinates (x',y') of a world point, the aboverelationships constitute two equations in three unknowns (x, y, and z)and are thus not sufficient to recover the 3D coordinates of thecorresponding world point, W. Referring to FIG. 7, the informationobtained from n single image 601a from a first vantage point 600adefines a ray 605a in 3D space originating at the image point C_(a),passing through the lens center 600a, and extending to infinity. Bydefinition, the actual world point W lies somewhere on this ray, butadditional information is needed to determine its exact location. If asecond image 601b, taken from a second vantage point 600b (whoseposition and orientation with respect to the first vantage point isknown), is available, then the corresponding image point C_(b) in thesecond image and the location of the second vantage point 600b define asecond ray 605b in space, such that the world point W lies on this ray awell. Using known mathematical techniques, the two rays can be resolvedin the same coordinate system and their intersection can be computed,giving the 3D world coordinates (x,y,z) of the point W.

Most camera lenses introduce distortions which causes the correspondenceof world and image points to depart from the above pin-hole model. Theprocess of calibrating the camera/lens system can estimate the natureand amount of such distortions and the resulting mathematical model canbe used to effectively "undistort" the image points. The pin-hole cameramodel can then be applied to the undistorted image. A number oftechniques for calibrating camera/lens systems are known.

As part of the interaction with a two-dimensional image of the patient'sanatomy displayed to the surgeon on a conventional monitor, the surgeonmay wish to designate (i.e., point to) a particular image locationwithin the displayed image. The surgeon may point to a particular imagelocation by using any of the following means: (a) by positioning asurgical instrument equipped with a distinct and clearly visible visualtarget so that the image of the visual target on the display coincideswith the desired image location, (b) by manipulating a graphical objecton the screen using an input device mounted on a surgical instrument(such as joystick 268 in FIGS. 1 and 3 or a similar device), or (c) bymanipulating a graphical object on the screen using a conventionalmouse. In method (a) the visual target may consist of a brightly coloredspot or a known geometric pattern of such spots at a known position onthe instrument (e.g., pattern 276 in FIGS. 1 and 3). The use of a brightcolor, distinct from any color naturally occurring inside the patient'sbody, greatly simplifies the problem of locating artificial visualtargets and lessens the chances of erroneous location of such targets.Such spots on the surgical instrument can be located using known imageprocessing techniques, involving thresholding (to isolate the spots fromthe rest of the image) and computationally determining the centers ofthe so obtained thresholded regions. In methods (b) and (c) the positionof the feature of interest is taken as the final position of thegraphical object.

Once the 2D coordinates of an image location have been specified tocomputer 243, the computer can confirm the location by marking thelocation with a graphical object superimposed on the image. In oneembodiment of this method of confirming an image location to thesurgeon, 2D cross-hair cursors or 2D box cursors can be used to show thelocation of interest in the image. The "image", in this context, can beeither a TV camera image or a computer generated graphical rendition ofthe anatomical area of interest.

We have so far described a variety of method for the surgeon to specifya particular 2D location of interest in a monoscopic image. We nextdiscuss methods, such as image processing, to determine positionalinformation about three-dimensional anatomical features and/or surgicalinstruments in the patient's body.

Referring to FIGS. 1 and 3, if a stereoscopic display (live or static)of the patient's anatomy is available during the surgical procedure,then a surgeon can designate the desired 3D anatomical feature ofinterest by manipulating a 3D stereoscopic graphical object (cursor) onthe stereoscopic display 272 until the graphical object is coincidentwith the desired anatomical feature. Any of the appropriateaforementioned input devices and modalities 248 (such as the surgicaltool mounted joystick or trackball, voice, etc.) can be used to specifythe desired motion of the graphical object within the stereoscopicvolume of the image.

If the actual physical size of a designated object is known, itsdistance from the viewing instrument may be estimated from the size ofits image, as seen by the viewing instrument. Since we know that thefeature lies on a ray originating at the center of inn age of thefeature and passing through the vantage point as shown in FIG. 7, theposition of the feature relative to the viewing instrument may then becomputed. Let the size of the feature in the image be l, let thecorresponding actual size of the feature be s, and let f denote thefocal length of the camera. The distance z from the camera lens to thefeature of interest can then be computed as z=(f×s)/l.

Referring to FIG. 8, in one embodiment, where passive visual targets 701on a surgical instrument 700 are used, the position of a 3D feature(e.g., a surgical instrument 700) can be determined as follows: At leastthree non collinear circular spots 701 of known diameter s are marked onthe surgical instrument 700 (FIG. 8a). Since the surgical instrument mayhave an arbitrary orientation with respect to the camera, these spotswill in general appear on the image plane as ellipses 705 (FIG. 8b).However, the length of the major axis of each ellipse l will be the sameas the diameter of the circular image that would be seen if thecorresponding circular spot were presented at that same distance fromthe lens in such a manner that the plane in which it lies isperpendicular to the view axis of the camera. Let the length of themajor axis of the observed ellipse as it appears in the image be l (FIG.8b). Then the distance of the spot from the camera lens can be computedfrom z=(f×s)/l. Having performed this computation for at least threespots and knowing the position of the spot pattern with respect to thetip of the surgical instrument suffices to compute the 3D location ofthe tip of the surgical instrument with respect to the camera. Othertechniques, known in the art, permit calculation of the position andorientation, relative to the camera, of a pattern of five dots from the2D positions of their centroids in the image obtained. Other patterns ofdots or other visual targets can be used as well. The 3D location of thetip of the instrument relative to the camera may then be readilycomputed from the known position of the tip relative to the visualtarget.

Additionally, stereo image processing may be used to precisely locate 3Danatomical features. In one embodiment, image processing can be used inconjunction with a stereoscopic camera to locate an anatomical feature.Referring to FIG. 3, surgical instrument 254a is a stereoscopic medicalcamera, comprising of two independent lens systems or optical fibers andis capable of transmitting two simultaneous images from the patient'sbody. The lenses are separated by a small (known) distance d, as shownin FIG. 9. The 3D position of the anatomical feature relative to thecamera tip can be computed from the pin-hole camera model (FIG. 6).Specifically, if the image plane locations of the center of the featureof interest in the two images are denoted by f₁ =(x₁, Y₁) and f₂ =(x₂,y₂), as shown in FIG. 9, then the distance z of the feature center fromthe camera lens can be computed as z=(f×d)/c, where c=√(x₂ -x₁)² +(y₂-y₁)² and f denotes the focal length of the camera. Image correlationtechniques or other image processing techniques known to the art may beused to locate features in images.

Referring again to FIG. 9, in another embodiment, using only a monocularcamera, image processing techniques can be used to determine theposition of an anatomical feature in three dimensions as follows: Afirst image 601a of the anatomical feature is acquired and a referencerepresentation (such as a multi-resolution image pyramid representationknown in the image processing art) is stored. The manipulator 242 isused to displace the camera lens tip 600a laterally by a known amount d,and a second image 601b is acquired. The center of the feature ofinterest W is located in the second image, using the referencerepresentation of the feature, by means of correlation techniques (suchas multi-resolution normalized correlation methods known in the art) andthe 3D displacement of the anatomical feature from the camera tip may becomputed as in the case above. Specifically, if the image planelocations of the feature of interest W in the two images 601a and 601bare denoted by f₁ =(x₁, y₁) and f₂ =(x₂, y₂), respectively, then thedistance z of the feature from the camera lens can be computed asz=(f×d)/c, where c=√x₂ -x₁)² +(y₂ -y₁)² and f denotes the focal lengthof the camera.

In another embodiment, the physical constraint of maintaining minimaltranslational motion of the telescope with respect to the port of entryinto the patient's body may preclude laterally displacing the telescopeto obtain a second image, as described above. Referring to FIG. 7, inthis embodiment, a first image 601a is obtained from the first vantagepoint and the center W of the feature of interest is located in theimage at image location C_(a) The telescope is then rotated by a small(known) amount about the port of entry, such that the desired feature isstill within the field of view of the telescope, and a second image 601bis obtained. Note that the second vantage point has a different positionand orientation than the first vantage point. The feature center W islocated in the second image at image location C_(b). The 3D position-ofthe feature center W is then obtained by computing the intersection ofthe rays 605a and 605b, as described previously. As above, imagecorrelation techniques or other image processing techniques known to theart may be used to locate features in images. Alternatively, the surgeonmay be asked to manually designate the image location of the featurecenter in the two images using any of the means of designating imagelocations described previously.

Once a 3D feature has been designated and its 3D location successfullycomputed, computer 243 can confirm its location by marking the locationwith a 3D stereoscopic graphical object superimposed on the stereoscopicimage of the area of interest. In one embodiment of this method ofconfirming 3D feature location to the surgeon, 3D cross-hair cursors or3D box cursors can be used to show the feature's 3D location within thestereoscopic view volume. The "image", in this context, can be either aTV camera image or a computer generated graphical rendition of theanatomical area of interest.

Once the 3D positions of anatomical features are stored in computer 243,this information may be used to control the position and orientation ofthe camera tip relative to the features so as to provide any desiredfield of view.

Referring to FIG. 1, in one mode the surgeon can designate a first and asecond 2D location in the image, using any of the means for designating2D locations discussed above. The surgeon can then instruct themanipulator 242 (using any appropriate input device or modality asdescribed earlier) to reposition the camera tip 266 so that theanatomical feature f₁, whose image appeared at the first 2D imagelocation prior to camera motion, appears at the second 2D location inthe image after the camera motion. The distance of the camera tip 266from the anatomical feature f₁ remains constant during the cameramotion. A special case of this mode is the case where the second 2Dlocation is the center of the image. In this case the camera isrepositioned so that the anatomical feature f₁ appears to move to thecenter of the displayed image, i.e., the camera is "centered" over theanatomical feature.

In another mode the surgeon can specify a sequence of 2D locations in animage and instruct the manipulator 242 to move the camera tip 266, at aconstant elevation, so that the camera traverses the path defined by thesequence of 2D locations in the image. In one embodiment, this sequenceof image locations can correspond to image locations of distinct smallanatomical features within the camera's field of view. In anotherembodiment, the sequence of image locations can correspond to imagelocations of a boundary of a large anatomical feature, such as a bloodvessel. This mode of repositioning the camera can be viewed also asspecifying the desired apparent motion of an anatomical feature(corresponding to the last 2D location in the specified sequence) withrespect to the image. The term "apparent motion of an anatomicalfeature" is used to emphasize that the anatomical feature does notphysically move, but only appears to move relative to the image due tothe motion of the camera. Specifically, the execution of this modeproceeds as follows: The sequence of 2D image locations is processed bycomputer 243 into a continuous path by the process of interpolation. Thecamera is then centered over the anatomical feature corresponding to thefirst designated 2D image location as described in the previousparagraph. The camera is then repeatedly positioned so as to center eachof the successive interpolated 2D locations within its field of view,thereby effectively traversing the path as defined by the surgeon. Thesurgeon directly controls both the direction and speed of the cameramotion by means of the surgical tool mounted joystick or any otherappropriate input means.

In another mode the surgeon can specify an increment of motion along thecamera's axis of view and reposition the camera along this axis by thedesignated amount. The "axis of view" in this context is defined as theline joining the camera lens center and the point p on the patient'sanatomy which appears in the center of the camera image. This modeeffectively implements a zoom function with respect to a 3D anatomicalfeature, where the zoom factor (i.e., desired enlargement or contractionof the image of the anatomical feature) is specified by the surgeoninteractively. In particular, this mode can be implemented by allowingthe surgeon to interactively manipulate a graphical cursor on the screenwhereby he can specify the desired zoom factor by enlarging orcontracting one such cursor with respect to a reference cursor whosesize does not change during the zoom factor specification. Anyappropriate input device 248 can be used to manipulate the cursorobject. Computer 243 then uses the relative geometry of the two cursorsto compute the direction and magnitude of the camera motion increment,which is necessary to effect the specified zoom factor. Alternatively,voice input can be used to specify the zoom factor. Once the cameramotion increment has been computed, computer 243 instructs manipulator242 to (slowly) reposition the camera tip 266 by that amount along theaxis of view, thereby obtaining the desired zoom factor. Note that thepoint p, as defined above, remains at the center of the image throughoutthe zooming process.

In another mode, the surgeon can directly control a desired direction ofmotion of the camera vantage point via an instrument-mounted inputdevice. In the preferred embodiment, this input device is a sixdegree-of-freedom joystick. Using such a joystick, the surgeon can thenarbitrarily reposition and reorient the camera in all six degrees offreedom simultaneously. By selecting different subsets of the full sixdegree-of-freedom motion, a number of useful control modes can beimplemented. In particular, if the translational controls of the sixdegree-of-freedom joystick are disabled or only a threedegree-of-freedom input device is available, a camera motion controlmode can be implemented, where the camera tip is constrained to movealong the surface of an imaginary sphere, centered at the currentanatomical feature of interest and having radius equal to the currentdistance of the camera tip from the feature. In another embodiment,where only a two degree-of-freedom input device is available, any two ofthe six degrees of freedom can be controlled by the device at any giventime. For instance, pressing a two degree-of-freedom joystick in thedirection toward the tip of the instrument on which the joystick ismounted can be interpreted to mean "zoom in", and pressing away from thetip can mean "zoom out". Releasing the joystick can mean "stop".Similarly, exerting pressure or force on a two degree-of-freedomjoystick in a direction perpendicular to the long axis of the camera canbe interpreted by computer 243 to mean a desired lateral motion of thecamera at the current elevation in the direction of the exertedpressure. Additionally, the velocity of the camera motion can be madeproportional to the amount of exerted pressure on the joystick.

In another mode the surgeon can manipulate a graphical objectsuperimposed on the image of the patient's anatomy to specify a desiredview of a particular feature of interest. The camera is thenautomatically positioned to achieve the desired view. A particularimplementation of this mode would proceed as follows: An image of thepatient's anatomy is obtained and displayed to the surgeon on a displaymonitor. The surgeon is then allowed to designate a feature of interestin a 2D or 3D image, unless the desired feature has already beendesignated and is visible. Next the surgeon can interactively manipulatea graphical object (e.g., cursor, slider, etc.) superimposed on theimage of the patient's anatomy on the display screen to specify thedesired view of the feature of interest. For example, the viewspecification could specify the desired vantage point of the cameraanywhere on the surface of a sphere of a given radius centered at thefeature of interest. Computer 243 then computes the appropriatedisplacement of the camera and instructs the manipulator 242 to executethe motion, thereby obtaining the desired view of the feature ofinterest.

If the surgical augmentation system comprises two independentlycontrolled robotic systems, as illustrated in FIG. 4, another mode ofusing the 3D positional information about the patient's anatomicalfeatures to reposition a surgical instrument can be used, where theinstrument being repositioned is the second surgical instrument, ratherthan the surgical telescope. In one embodiment of this invention, thesecond surgical instrument could be surgical forceps, which arerepositioned such that the jaws of the instrument are coincident withthe current 3D anatomical feature and a tissue sample of this featurecan thus be obtained by closing the instrument's jaws.

Referring to FIG. 10, the capability of interactively designating thedesired view of a particular 3D feature of interest and letting thecomputer compute the resulting new location of the medical telescope isespecially important in situations where the telescope's optics providea lateral, rather than a straight-ahead (α=0°) view. Telescopes with thedirection-of-view anywhere between 30° and 135° (with respect to theinstrument's long axis) are commonly used in laparoscopic and similarprocedures. FIG. 10 illustrates a telescope with the direction-of-viewof α=45°. Manually positioning such a telescope to achieve a desiredview can be extremely difficult even for an experienced camera operatoras the relative transformations between the telescope, the patient'sanatomy and the image coordinates become rather complex and unintuitive.However, adding a single rigid body transformation to the computationalchain in the computer software accounts for the fact that thedirection-of-view is different from 0°. In a particular implementation,a coordinate frame F_(c) is associated with a 0° telescope, and thecomputer keeps track of the rigid body transformations between themanipulator, the camera, and the various anatomical features ofinterest. The mathematical methods and techniques of representing andmanipulating rigid body transformations are well known to the art ofrobotics and computer graphics. Camera motions needed to effect aparticular zoom Factor, for example, are then computed relative to thiscamera frame F_(c). For the case of a non-straight telescope, such asthe telescope in FIG. 10, a new coordinate frame F_(c) ' is defined byrotating the frame F_(c) through an angle of -α about a line passingthrough the center of the lens tip and parallel to the X-axis of thecoordinate frame F_(c). The rigid body transformation ^(c) T_(c')relating the new camera frame F_(c') to the default, 0° location of thecamera frame F_(c), is used to account for the non-zero direction ofview of the telescope. Using the transform F_(c') in place of F_(c) inthe computation of the new desired position of the telescope for aparticular desired view now results in correct repositioning of thetelescope regardless of its direction-of-view.

The visual information transmitted from the patient's body andoptionally augmented by image processing and computer graphics can bedisplayed to a surgeon in a number of ways.

Referring to FIG. 1, in one mode of information display, the images ofthe patient's anatomy can be displayed to the surgeon as a combinationof live and still images (a live image is an image obtained from thecamera that is continuously updated with new information, whereas astill image is not). In one embodiment of this mode, the image to bedisplayed on the monoscopic monitor 247 is produced as follows: Awide-angle monoscopic image of the patient's anatomy is obtained usingthe surgical instrument 254 and displayed on the monitor 247 as a staticimage. The camera is then zoomed in For a closer view of the currentfeature off interest and a portion of this live TV image is displayedsuperimposed on top of the static wide-angle image. The staticmonoscopic view of the overall area of interest thus provides contextualinformation about the patient's anatomy under observation, whereas thelive subimage shows a magnified detail area surrounding the currentanatomical feature of interest.

In an alternative embodiment of this display mode, the static wide-anglecontextual information can be a computer-graphic rendering of thepatient's anatomy. This graphical information can be derived fromcomputer models of the patient's anatomy constructed on the basis of theinformation gathered during preoperative imaging and scanning. Asbefore, a portion of the image surrounding the current anatomicalfeature of interest is replaced with a live magnified TV image of thisarea. Here, the computer generated image and actual live TV image aremerged into a single display image and must thus be properly registeredwith respect to each other to ensure proper correspondences ofanatomical points and features between the two images. A number oftechniques for achieving registration between images are known to theart. In the simplest embodiment, the 3D locations of a number of knownanatomical landmarks represented in the computer model would beidentified by 3D image processing techniques. The 3D locations of theselandmarks can then be used to compute the appropriate perspective viewfor displaying the graphical model.

In another embodiment of this display mode, the static wide-anglecontextual information can be a computer-graphic rendering of thepatient's anatomy, as above. Similarly, a portion of the imagesurrounding the current anatomical feature of interest is replaced witha live magnified TV image of this area, as above. In addition, the liveTV image of the area of detail can be augmented by superimposing staticedge information, which can be derived either from a computer graphicsmodel or as a result of image processing (edge extraction) on the TVimage. The advantage of this display mode is that the superimposed edgeshighlight the ongoing changes within the area of detail reflected in thelive TV image with respect to the previous (static) appearance of thisarea.

In another embodiment of this mode of displaying information to thesurgeon, the static wide-angle view of the overall area of interest canbe displayed as a static stereoscopic image. Referring to FIG. 1, thisis achieved as follows: A static image of the overall area of interestis obtained from a first vantage point using the surgical instrument 254and camera 259. The tip of the camera lens 266 is then displaced by asmall known amount and a second static image of the area of interest istaken from this displaced vantage point. The two images are then fed asinput to the stereo display system 271 and displayed on the steroscopicmonitor 272 as a static stereoscopic wide-angle view of the overallanatomical area of interest. In some cases where only the distalmanipulator 240 is moved to displace the camera, there may be some smallangular misalignment of the two images so obtained. Experiment has shownthat this misalignment can often be ignored, since the human visualsystem is very adept at fusing slightly misaligned images.Alternatively, the misalignment can be largely compensated for by usingimage transformation techniques known in the art. Next, the camera iszoomed in for a close-up view of the current anatomical feature ofinterest and a portion of the static wide-angle image is replaced by themagnified live monoscopic view of the anatomical feature of interest, asbefore. This results in an image, where the overall contextualinformation is a static stereoscopic image, providing the surgeon with asense of the global three-dimensional relationships within the viewingvolume, and the area surrounding the current anatomical feature ofinterest, where the surgeon's concentration is focused, is magnified anddisplayed as a live monoscopic image.

In a modification of the above mode of display, the live TV image of thearea of detail can be augmented by superimposing static edgeinformation, which can be derived either from a computer graphics modelor as a result of image processing (edge extraction) on the TV image. Asdescribed previously, the advantage of this display mode is that thesuperimposed edges highlight the ongoing changes within the area ofdetail reflected in the live TV image with respect to the previous(static) appearance of this area.

Referring to FIG. 3, another embodiment of the present inventionregarding display of visual information to the surgeon, uses thestereoscopic medical camera 254a to obtain a static stereoscopicwide-angle image of the overall anatomical area of interest. Then, asabove, the stereoscopic camera is zoomed in closer over the current 3Danatomical feature of interest and a portion of the static imagesurrounding the feature of interest is replaced by a magnified livestereoscopic TV image as transmitted from the patient's body by cameras259a and 259b.

In order to emphasize the changes occurring within the area of detail,edge information corresponding to a previous state of the area of detailcan be superimposed on the live stereoscopic image, as before.

Again referring to FIG. 3, another embodiment of the present inventionuses the stereoscopic medical camera 254a in conjunction withstereoscopic computer graphics to provide a display of the patient'sanatomy. In this embodiment, the static stereoscopic view of the overallanatomical area of interest is derived from computer models of thepatient's anatomy and displayed on the monitor 272 as a 3D stereoscopicgraphical image via the stereo display system 271. As above, thestereoscopic camera is then zoomed in closer over the current 3Danatomical feature of interest and a portion of the static graphicalimage surrounding the feature of interest is replaced by a magnifiedlive stereoscopic TV image as transmitted from the patient's body bycameras 259a and 259b.

Again, in order to emphasize the changes occurring within the area ofdetail, edge information corresponding to a previous state of the areaof detail can be superimposed on the live stereoscopic image, as before.

Referring to FIG. 1, another mode of display of anatomical informationto a surgeon uses the monoscopic camera 254 to provide the surgeon witha live stereoscopic image of the patient's anatomy. In this mode, theinformation supplied to one of the surgeon's eyes is derived onecomputer models of the patient's anatomy and is displayed as a graphicalimage computed from the vantage point displaced a small known distancelaterally from the current vantage point of the surgical instrument 254.The information supplied to the other eye is the live image of thepatient's anatomy as provided by the camera 259 attached to the surgicalinstrument 254. In this mode, one eye therefore receives static computergenerated view of the patient's body, whereas the other eye receives alive image transmitted by the camera from a slightly displaced vantagepoint. If the computer-graphic model is properly registered with theactual anatomy, the human brain will fuse the two images into a proper3D stereoscopic image.

In another embodiment of the above mode of display of anatomicalinformation to the surgeon, image processing is used in conjunction withlive video information to produce a live stereoscopic display to thesurgeon. Referring to FIG. 1, in this embodiment of the presentinvention, a first image of the patient's anatomy under observation isobtained and transferred to the image processing system 245. The cameratip 266 is then displaced laterally a small known amount and a secondimage is obtained from this second vantage point and transferred to theimage processing system 245. The image processing system and known imageprocessing techniques are then used to extract edge information from thetwo images. A stereoscopic display is then produced by supplying thestereo display system 271 with only edge information in one of the inputchannels (left/right eye) and a live video signal with overlaid edgeinformation in the other input channel (right/left eye). Subsequently,only information to one of the two eyes is updated with live video astransmitted by camera 259. This provides enough information for thehuman brain to "fill in" the missing information and interpret the imageas a proper stereoscopic 3D image.

Alternatively, a display mode as above can be used, where the edgeinformation is not obtained by image processing, but rather derived froma computer graphical model of the patient's anatomy.

Aside from visual information, the surgeon can receive non-visualinformation about the locations of features or the general state of thesystem as well. One non-visual channel of communication between thesurgeon and the system is the voice recognition and speech synthesissubsystem (267, FIG. 1). For example, synthesized voice messages can beissued by the system to inform the surgeon of the exact location of hissurgical instrument with respect to an anatomical feature of interest.Likewise, synthesized messages confirming successful receipt of a voicecommand can be used to assure the surgeon that the system correctlyinterpreted his command(s). General system state or change of systemstate information can be relayed to the surgeon using synthesized voiceas well. An example of this would be a synthesized speech message to thesurgeon stating the exact distance by which the camera was moved duringa zooming operation.

An alternative method of relaying non-visual information to the surgeonis tactile feedback. In one embodiment of this invention, tactilefeedback conveyed to the surgeon through a hand-held orinstrument-mounted input device (such as a joystick) can be used toalert the surgeon that he has positioned a graphical object or asurgical instrument in the vicinity of the current anatomical feature ofinterest. The tactile feedback can be delivered to the surgeon's hand orfinger (whichever is in contact with the joystick) by instrumenting thejoystick control with a computer controlled vibrator. When the vibratoris activated by the computer, the joystick control starts vibrating withappropriate frequency and amplitude, such that the oscillations arereadily discernible by the surgeon, but do not distract him from hispositioning task or otherwise interfere with his work.

We claim:
 1. A method determining positional information about ananatomical feature within a patient's body comprising the stepsof:inserting a first surgical instrument into the patient's body, theinstrument having a means for transmitting an image out of the patient'sbody; designating an anatomical feature of interest by pointing with asecond surgical instrument, one or more visual targets being on thesecond instrument; transmitting an image of the designated anatomicalfeature out of the patient's body using the first surgical instrument;determining positional information about the designated anatomicalfeature of interest by using image processing.
 2. A method ofdetermining positional information about an anatomical feature within apatient's body, comprising the steps of:inserting a first surgicalinstrument into the patient's body, the instrument having a means fortransmitting an image out of the patient's body; transmitting an imageof the designated anatomical feature out of the patient's body using thefirst surgical instrument; displaying the transmitted image; designatingan anatomical feature of interest by manipulating a computer generatedgraphics object displayed and superimposed on the display imagetransmitted by the first surgical instrument; and determining positionalinformation about the designated anatomical feature of interest by usingimage processing.
 3. A method of determining positional informationabout an anatomical feature, as in claim 2, where the computer generatedgraphics object is manipulated by the use of a joystick mounted on thefirst surgical instrument.
 4. A method of determining positionalinformation about an anatomical feature, as in claim 2, where thecomputer generated graphics object is manipulated by the use of a forcesensing device mounted on the first surgical instrument.
 5. A method ofdetermining positional information about an anatomical feature within apatient's body, comprising the steps of:inserting a first surgicalinstrument into the patient's body, the instrument having a means fortransmitting an image out of the patient's body; transmitting an imageof the designated anatomical feature out of the patient's body using thefirst surgical instrument; designating an anatomical feature of interestby using either one of the following steps (1) and (2):(1) pointing witha second surgical instrument having a visual target and (2) manipulatinga computer generated graphics object displayed and superimposed on theimage transmitted by the first surgical instrument; and furthercomprising the steps of: determining positional information about thedesignated anatomical feature of interest by using image processing; andproviding the positional, information to a surgeon.
 6. A method ofdetermining positional information about an anatomical feature, as inclaim 5, where the information is provided to the surgeon in the form ofsynthesized speech.
 7. A method of determining positional informationabout an anatomical feature, as in claim 5, where the information isprovided to the surgeon in the form of tactile feedback.
 8. A method ofdetermining positional information about an anatomical feature, as inclaim 7, where the tactile feedback is provided in the form ofvibrations on one of the surgical instruments held by the surgeon.
 9. Amethod of determining positional information about an anatomicalfeature, as in claim 5, where the information is provided to the surgeonin the form of a computer generated graphics object superimposed on theimage obtained from the first surgical instrument.
 10. A method ofdetermining positional information about an anatomical feature, as inclaim 9, where the computer graphics object is displayed in twodimensions.
 11. A method of determining positional information about ananatomical feature, as in claim 9, where the computer graphics object isdisplayed in three dimensions.
 12. A method of determining positionalinformation about an anatomical feature comprising:inserting a first anda second surgical instrument into the patient's body, each instrumenthaving a means for transmitting an image out of the patient's body;obtaining a first image of the feature from a first vantage point usingthe first surgical instrument; obtaining a second image of the featureusing the second surgical instrument from a second vantage point, thesecond vantage point being at a known position and orientation withrespect to the first vantage point; locating the anatomical feature inboth images; computing the position of the anatomical feature relativeto the vantage points using positional information about the feature ineach image, together with the known position and orientation of the twovantage points with respect to each other.
 13. A method of determiningpositional information about an anatomical feature, as in claim 12,where the anatomical feature is located in at least one of the images bycomputer image processing.
 14. A method of determining positionalinformation about an anatomical feature, as in claim 12, where the firstvantage point is the position of the first surgical instrument, beingone lens of a stereoscopic camera, and the second vantage point is theposition of the second surgical instrument, being the position of thesecond lens of a stereoscopic camera.
 15. A method of determiningpositional information about an anatomical feature comprising the stepsof:inserting a first surgical instrument into the patient's body, theinstrument having a means for transmitting an image out of the patient'sbody; obtaining a first image of the feature from a first vantage pointusing the first surgical instrument; obtaining a second image of thefeature using the first surgical instrument from a second vantage point,the second vantage point being at a known position and orientation withrespect to the first vantage point; locating the anatomical feature inthe first and second images; and computing the position of theanatomical feature relative to the first and second vantage points usingpositional information about the feature in each of the first and secondimages together with the known position and orientation of the twovantage points with respect to each other.